<!DOCTYPE html>
<html lang=zh-CN>
<head>
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="google-site-verification" content="xBT4GhYoi5qRD5tr338pgPM5OWHHIDR6mNg1a3euekI" />
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <meta property="og:description" content="samuel-24276的博客">
    <meta property="og:type" content="website">
    <meta name="description" content="samuel-24276的博客">
    <meta name="keyword"  content="aircloud">
    <link rel="shortcut icon" href="/myblog/img/favicon.ico">

    <title>
        
        高德地图-中国主要城市交通健康榜-爬虫项目 - samuel-24276的博客 | samuel-24276&#39;s Blog
        
    </title>

    <!-- Custom CSS -->
    
<link rel="stylesheet" href="/myblog/css/aircloud.css">

    
<link rel="stylesheet" href="/myblog/css/gitment.css">

    <!--<link rel="stylesheet" href="https://imsun.github.io/gitment/style/default.css">-->
    <link href="//at.alicdn.com/t/font_620856_28hi1hpxx24.css" rel="stylesheet" type="text/css">
    <!-- ga & ba script hoook -->
    <script></script>
<meta name="generator" content="Hexo 7.1.1"></head>

<body>

<div class="site-nav-toggle" id="site-nav-toggle">
    <button>
        <span class="btn-bar"></span>
        <span class="btn-bar"></span>
        <span class="btn-bar"></span>
    </button>
</div>

<div class="index-about">
    <i> Like what you do, do what you like </i>
</div>

<div class="index-container">
    
    <div class="index-left">
        
<div class="nav" id="nav">
    <div class="avatar-name">
        <div class="avatar ">
            <img src="/myblog/img/avatar.jpg" />
        </div>
        <div class="name">
            <i>samuel-24276</i>
        </div>
    </div>
    <div class="contents" id="nav-content">
        <ul>
            <li >
                <a href="/myblog/">
                    <i class="iconfont icon-shouye1"></i>
                    <span>主页</span>
                </a>
            </li>
            <li >
                <a href="/myblog/tags">
                    <i class="iconfont icon-biaoqian1"></i>
                    <span>标签</span>
                </a>
            </li>
            <li >
                <a href="/myblog/archives">
                    <i class="iconfont icon-guidang2"></i>
                    <span>存档</span>
                </a>
            </li>
            <li >
                <a href="/myblog/collect/">
                    <i class="iconfont icon-shoucang1"></i>
                    <span>收藏</span>
                </a>
            </li>
            <li >
                <a href="/myblog/about/">
                    <i class="iconfont icon-guanyu2"></i>
                    <span>关于</span>
                </a>
            </li>
            
            <li>
                <a id="search">
                    <i class="iconfont icon-sousuo1"></i>
                    <span>搜索</span>
                </a>
            </li>
            
        </ul>
    </div>
    
        <div id="toc" class="toc-article">
    <ol class="toc"><li class="toc-item toc-level-1"><a class="toc-link" href="#1-%E5%9B%BE%E5%BD%A2%E5%8C%96%E6%95%B0%E6%8D%AE%E7%95%8C%E9%9D%A2%E5%B1%95%E7%A4%BA"><span class="toc-text">1.图形化数据界面展示</span></a><ol class="toc-child"><li class="toc-item toc-level-2"><a class="toc-link" href="#1-1-%E5%88%9B%E5%BB%BA%E5%B9%B6%E6%B3%A8%E5%86%8C%E5%AD%90%E5%BA%94%E7%94%A8"><span class="toc-text">1.1.创建并注册子应用</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#1-2-%E9%85%8D%E7%BD%AEMySQL"><span class="toc-text">1.2.配置MySQL</span></a><ol class="toc-child"><li class="toc-item toc-level-3"><a class="toc-link" href="#1-2-1-%E5%AE%89%E8%A3%85%E9%A9%B1%E5%8A%A8%E7%A8%8B%E5%BA%8F"><span class="toc-text">1.2.1.安装驱动程序</span></a></li><li class="toc-item toc-level-3"><a class="toc-link" href="#1-2-2-%E8%B0%83%E7%94%A8%E6%95%B0%E6%8D%AE%E5%BA%93"><span class="toc-text">1.2.2.调用数据库</span></a></li><li class="toc-item toc-level-3"><a class="toc-link" href="#1-2-3-%E4%BF%AE%E6%94%B9DATABASE%E9%85%8D%E7%BD%AE%E4%BF%A1%E6%81%AF"><span class="toc-text">1.2.3.修改DATABASE配置信息</span></a></li><li class="toc-item toc-level-3"><a class="toc-link" href="#1-2-4-%E5%88%9B%E5%BB%BA%E6%95%B0%E6%8D%AE%E5%BA%93"><span class="toc-text">1.2.4.创建数据库</span></a></li></ol></li><li class="toc-item toc-level-2"><a class="toc-link" href="#1-3-ORM%E6%A1%86%E6%9E%B6%E6%98%A0%E5%B0%84"><span class="toc-text">1.3.ORM框架映射</span></a><ol class="toc-child"><li class="toc-item toc-level-3"><a class="toc-link" href="#1-3-1-mysql"><span class="toc-text">1.3.1.mysql</span></a></li><li class="toc-item toc-level-3"><a class="toc-link" href="#1-3-2-%E6%95%B0%E6%8D%AE%E5%BA%93%E5%AD%98%E5%82%A8%E5%86%85%E5%AE%B9%E5%88%86%E6%9E%90"><span class="toc-text">1.3.2.数据库存储内容分析</span></a></li><li class="toc-item toc-level-3"><a class="toc-link" href="#1-3-3-ORM"><span class="toc-text">1.3.3.ORM</span></a></li></ol></li><li class="toc-item toc-level-2"><a class="toc-link" href="#1-4-%E9%85%8D%E7%BD%AEadmin%E7%AB%99%E7%82%B9"><span class="toc-text">1.4.配置admin站点</span></a><ol class="toc-child"><li class="toc-item toc-level-3"><a class="toc-link" href="#1-4-1%E5%90%AF%E5%8A%A8%E6%9C%8D%E5%8A%A1"><span class="toc-text">1.4.1启动服务</span></a></li><li class="toc-item toc-level-3"><a class="toc-link" href="#1-4-2-%E5%9C%A8%E7%BB%88%E7%AB%AF%E5%88%9B%E5%BB%BA%E7%94%A8%E6%88%B7"><span class="toc-text">1.4.2.在终端创建用户</span></a></li><li class="toc-item toc-level-3"><a class="toc-link" href="#1-4-3-%E9%85%8D%E7%BD%AEadmin-py%E6%96%87%E4%BB%B6"><span class="toc-text">1.4.3.配置admin.py文件</span></a></li></ol></li></ol></li><li class="toc-item toc-level-1"><a class="toc-link" href="#2-%E9%AB%98%E5%BE%B7%E5%9C%B0%E5%9B%BE-%E4%B8%AD%E5%9B%BD%E4%B8%BB%E8%A6%81%E5%9F%8E%E5%B8%82%E4%BA%A4%E9%80%9A%E5%81%A5%E5%BA%B7%E6%A6%9C-%E7%88%AC%E8%99%AB%E9%A1%B9%E7%9B%AE"><span class="toc-text">2.高德地图-中国主要城市交通健康榜-爬虫项目</span></a><ol class="toc-child"><li class="toc-item toc-level-2"><a class="toc-link" href="#2-1-%E6%95%B0%E6%8D%AE%E6%80%BB%E5%85%A5%E5%8F%A3"><span class="toc-text">2.1.数据总入口</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#2-2-%E6%95%B0%E6%8D%AE%E5%88%86%E7%B1%BB"><span class="toc-text">2.2.数据分类</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#2-3-scrapy%E5%9F%BA%E6%9C%AC%E4%BD%BF%E7%94%A8"><span class="toc-text">2.3.scrapy基本使用</span></a><ol class="toc-child"><li class="toc-item toc-level-3"><a class="toc-link" href="#2-3-1-scrapy%E9%A1%B9%E7%9B%AE%E8%BF%87%E7%A8%8B"><span class="toc-text">2.3.1.scrapy项目过程</span></a></li><li class="toc-item toc-level-3"><a class="toc-link" href="#2-3-2-%E6%96%87%E4%BB%B6%E8%AF%B4%E6%98%8E"><span class="toc-text">2.3.2.文件说明</span></a></li><li class="toc-item toc-level-3"><a class="toc-link" href="#2-3-3-%E5%AE%8C%E5%96%84%E9%A1%B9%E7%9B%AE"><span class="toc-text">2.3.3.完善项目</span></a></li><li class="toc-item toc-level-3"><a class="toc-link" href="#2-3-4-%E6%95%B0%E6%8D%AE%E5%85%A5%E5%BA%93"><span class="toc-text">2.3.4.数据入库</span></a></li><li class="toc-item toc-level-3"><a class="toc-link" href="#2-3-5-%E7%88%AC%E5%8F%96%E6%95%B0%E6%8D%AE"><span class="toc-text">2.3.5.爬取数据</span></a></li></ol></li></ol></li></ol>
</div>
    
</div>


<div class="search-field" id="search-field">
    <div class="search-bg" id="search-bg"></div>
    <div class="search-container">
        <div class="search-input">
            <span id="esc-search"> <i class="icon-fanhui iconfont"></i></span>
            <input id="search-input"/>
            <span id="begin-search">搜索</span>
        </div>
        <div class="search-result-container" id="search-result-container">

        </div>
    </div>
</div>

        <div class="index-about-mobile">
            <i> Like what you do, do what you like </i>
        </div>
    </div>
    
    <div class="index-middle">
        <!-- Main Content -->
        


<div class="post-container">
    <div class="post-title">
        高德地图-中国主要城市交通健康榜-爬虫项目
    </div>

    <div class="post-meta">
        <span class="attr">发布于：<span>2020-05-18 22:14:38</span></span>
        
        <span class="attr">标签：/
        
        <a class="tag" href="/myblog/tags/#Python" title="Python">Python</a>
        <span>/</span>
        
        <a class="tag" href="/myblog/tags/#Scrapy" title="Scrapy">Scrapy</a>
        <span>/</span>
        
        <a class="tag" href="/myblog/tags/#Django" title="Django">Django</a>
        <span>/</span>
        
        
        </span>
        <span class="attr">访问：<span id="busuanzi_value_page_pv"></span>
</span>
</span>
    </div>
    <div class="post-content no-indent">
        <h1 id="1-图形化数据界面展示"><a href="#1-图形化数据界面展示" class="headerlink" title="1.图形化数据界面展示"></a>1.图形化数据界面展示</h1><p>创建应用<br><code>django-admin startproject GaodeMap</code></p>
<h2 id="1-1-创建并注册子应用"><a href="#1-1-创建并注册子应用" class="headerlink" title="1.1.创建并注册子应用"></a>1.1.创建并注册子应用</h2><p>创建子应用</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="built_in">cd</span> GaodeMap</span><br><span class="line">python manage.py startapp map</span><br><span class="line">python manage.py runserver</span><br></pre></td></tr></table></figure>

<p>注册子应用，在与项目同名的子文件夹下的settings.py文件下找到INSTALLED_APPS，加上一句：<br><code>&#39;map.apps.MapConfig&#39;,</code><br>查看文件结构<br><code>tree</code><br>文件目录说明：<br>admin.py:文件跟网站的后台管理站点配置有关。<br>apps.py:文件用于配置当前子应用的相关信息。<br>migrations:目录用于存放数据库迁移历史文件。<br>models.py:文件用户保存数据库模型类。<br>tests.py:文件用于开发测试用例，编写单元测试。<br>views.py:文件用于编写WEB应用视图。</p>
<h2 id="1-2-配置MySQL"><a href="#1-2-配置MySQL" class="headerlink" title="1.2.配置MySQL"></a>1.2.配置MySQL</h2><h3 id="1-2-1-安装驱动程序"><a href="#1-2-1-安装驱动程序" class="headerlink" title="1.2.1.安装驱动程序"></a>1.2.1.安装驱动程序</h3><p><code>pip install PyMySQL</code></p>
<h3 id="1-2-2-调用数据库"><a href="#1-2-2-调用数据库" class="headerlink" title="1.2.2.调用数据库"></a>1.2.2.调用数据库</h3><p>在Django工程同名子目录下_init_.py文件添加语句：</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> pymysql <span class="keyword">import</span> install_as_MySQLdb </span><br><span class="line">install_as_MySQLdb()</span><br></pre></td></tr></table></figure>
<p>作用是让Django的ORM能以mysqldb方式调用PyMySQL。</p>
<h3 id="1-2-3-修改DATABASE配置信息"><a href="#1-2-3-修改DATABASE配置信息" class="headerlink" title="1.2.3.修改DATABASE配置信息"></a>1.2.3.修改DATABASE配置信息</h3><p>在Django工程同名子目录下settings.py文件的DATABASES修改为：</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">DATABASES = &#123;</span><br><span class="line">    <span class="string">&#x27;default&#x27;</span>: &#123;</span><br><span class="line">        <span class="string">&#x27;ENGINE&#x27;</span>: <span class="string">&#x27;django.db.backends.mysql&#x27;</span>,</span><br><span class="line">        <span class="string">&#x27;HOST&#x27;</span>: <span class="string">&#x27;127.0.0.1&#x27;</span>,  <span class="comment"># 数据库主机</span></span><br><span class="line">        <span class="string">&#x27;PORT&#x27;</span>: <span class="number">3306</span>,  <span class="comment"># 数据库端口</span></span><br><span class="line">        <span class="string">&#x27;USER&#x27;</span>: <span class="string">&#x27;root&#x27;</span>,  <span class="comment"># 用户名</span></span><br><span class="line">        <span class="string">&#x27;PASSWORD&#x27;</span>: <span class="string">&#x27;root&#x27;</span>,  <span class="comment"># 你的密码</span></span><br><span class="line">        <span class="string">&#x27;NAME&#x27;</span>: <span class="string">&#x27;map&#x27;</span>   <span class="comment"># 数据库名字</span></span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure>

<h3 id="1-2-4-创建数据库"><a href="#1-2-4-创建数据库" class="headerlink" title="1.2.4.创建数据库"></a>1.2.4.创建数据库</h3><p>登录客户端：<br><code>mysql -uroot -p</code><br>显示所有数据库：<br><code>show databases;</code><br>创建数据库:<br><code>create database map default charset=utf8;</code></p>
<h2 id="1-3-ORM框架映射"><a href="#1-3-ORM框架映射" class="headerlink" title="1.3.ORM框架映射"></a>1.3.ORM框架映射</h2><h3 id="1-3-1-mysql"><a href="#1-3-1-mysql" class="headerlink" title="1.3.1.mysql"></a>1.3.1.mysql</h3><p>数据库、表、字段列、SQL语句</p>
<h3 id="1-3-2-数据库存储内容分析"><a href="#1-3-2-数据库存储内容分析" class="headerlink" title="1.3.2.数据库存储内容分析"></a>1.3.2.数据库存储内容分析</h3><p>城市编号，城市名字，拥堵延时指数cdi，高延时运行时间占比hlrr，拥堵路段里程比mrcr，平均车速speed，道路运行速度偏差率rrsdr</p>
<h3 id="1-3-3-ORM"><a href="#1-3-3-ORM" class="headerlink" title="1.3.3.ORM"></a>1.3.3.ORM</h3><p>编写models.py文件：</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> django.db <span class="keyword">import</span> models</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment"># Create your models here.</span></span><br><span class="line"><span class="comment"># 1.创建模型类</span></span><br><span class="line"><span class="keyword">class</span> <span class="title class_">MapInfo</span>(models.Model):</span><br><span class="line">    label = models.CharField(max_length=<span class="number">100</span>, null=<span class="literal">False</span>, verbose_name=<span class="string">&quot;城市编号&quot;</span>)</span><br><span class="line">    name = models.CharField(max_length=<span class="number">200</span>, default=<span class="string">&quot;无&quot;</span>, verbose_name=<span class="string">&quot;城市&quot;</span>)</span><br><span class="line">    cdi = models.CharField(max_length=<span class="number">200</span>, default=<span class="string">&quot;无&quot;</span>, verbose_name=<span class="string">&quot;交通拥堵指数&quot;</span>)</span><br><span class="line">    hlrr = models.CharField(max_length=<span class="number">200</span>, default=<span class="string">&quot;无&quot;</span>, verbose_name=<span class="string">&quot;高延时运行时间占比&quot;</span>)</span><br><span class="line">    mrcr = models.CharField(max_length=<span class="number">200</span>, default=<span class="string">&quot;无&quot;</span>, verbose_name=<span class="string">&quot;拥堵路段里程比&quot;</span>)</span><br><span class="line">    speed = models.CharField(max_length=<span class="number">200</span>, default=<span class="string">&quot;无&quot;</span>, verbose_name=<span class="string">&quot;平均速度&quot;</span>)</span><br><span class="line">    rrsdr = models.CharField(max_length=<span class="number">200</span>, default=<span class="string">&quot;无&quot;</span>, verbose_name=<span class="string">&quot;道路运行速度偏差率&quot;</span>)</span><br><span class="line">    <span class="string">&quot;&quot;&quot;</span></span><br><span class="line"><span class="string">    cdi = models.DecimalField(decimal_places=2, max_digits=20, default=&quot;0.00&quot;, verbose_name=&quot;交通拥堵指数&quot;)</span></span><br><span class="line"><span class="string">    hlrr = models.DecimalField(decimal_places=2, max_digits=20, default=&quot;0.00&quot;, verbose_name=&quot;高延时运行时间占比&quot;)</span></span><br><span class="line"><span class="string">    mrcr = models.DecimalField(decimal_places=2, max_digits=20, default=&quot;0.00&quot;, verbose_name=&quot;拥堵路段里程比&quot;)</span></span><br><span class="line"><span class="string">    speed = models.DecimalField(decimal_places=2, max_digits=20, default=&quot;0.00&quot;, verbose_name=&quot;平均速度&quot;)</span></span><br><span class="line"><span class="string">    rrsdr = models.DecimalField(decimal_places=2, max_digits=20, default=&quot;0.00&quot;, verbose_name=&quot;道路运行速度偏差率&quot;)</span></span><br><span class="line"><span class="string">    &quot;&quot;&quot;</span></span><br><span class="line"></span><br><span class="line">    <span class="keyword">class</span> <span class="title class_">Meta</span>:  <span class="comment"># 别名</span></span><br><span class="line">        verbose_name = <span class="string">&quot;全国主要城市最近七天内平均交通数据&quot;</span></span><br><span class="line">        verbose_name_plural = verbose_name</span><br><span class="line"></span><br><span class="line">    <span class="comment"># 魔法函数</span></span><br><span class="line">    <span class="keyword">def</span> <span class="title function_">__str__</span>(<span class="params">self</span>):</span><br><span class="line">        <span class="keyword">return</span> self.name</span><br><span class="line"></span><br><span class="line"></span><br></pre></td></tr></table></figure>

<p>终端操作 数据迁移</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">python manage.py makemigrations </span><br><span class="line">python manage.py migrate</span><br></pre></td></tr></table></figure>

<p>然后打开刚才的CMD命令窗口，输入：<br><code>use map;</code><br><code>show tables;</code><br>可以看到新增了许多迁移的表</p>
<h2 id="1-4-配置admin站点"><a href="#1-4-配置admin站点" class="headerlink" title="1.4.配置admin站点"></a>1.4.配置admin站点</h2><h3 id="1-4-1启动服务"><a href="#1-4-1启动服务" class="headerlink" title="1.4.1启动服务"></a>1.4.1启动服务</h3><p><code>python manage.py runserver</code><br>访问<code>127.0.0.1:8000/admin</code>网址<br>找到与工程同名子目录下的settings.py文件，修改最下方内容</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">LANGUAGE_CODE = <span class="string">&#x27;zh-Hans&#x27;</span></span><br><span class="line"></span><br><span class="line">TIME_ZONE = <span class="string">&#x27;Asia/Shanghai&#x27;</span></span><br></pre></td></tr></table></figure>
<h3 id="1-4-2-在终端创建用户"><a href="#1-4-2-在终端创建用户" class="headerlink" title="1.4.2.在终端创建用户"></a>1.4.2.在终端创建用户</h3><p><code>python manage.py createsuperuser</code><br>用户名：admin<br>邮箱：<a href="mailto:&#100;&#100;&#x64;&#49;&#50;&#51;&#64;&#49;&#54;&#51;&#46;&#x63;&#x6f;&#109;">&#100;&#100;&#x64;&#49;&#50;&#51;&#64;&#49;&#54;&#51;&#46;&#x63;&#x6f;&#109;</a><br>密码：admin123<br>找到网址<code>127.0.0.1:8000/admin</code>进行登录</p>
<h3 id="1-4-3-配置admin-py文件"><a href="#1-4-3-配置admin-py文件" class="headerlink" title="1.4.3.配置admin.py文件"></a>1.4.3.配置admin.py文件</h3><p>在创建的应用文件夹下找到admin.py进行配置</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">from</span> django.contrib <span class="keyword">import</span> admin</span><br><span class="line"></span><br><span class="line"><span class="comment"># Register your models here.</span></span><br><span class="line"><span class="keyword">from</span> django.contrib <span class="keyword">import</span> admin</span><br><span class="line"></span><br><span class="line"><span class="keyword">from</span> .models <span class="keyword">import</span> MapInfo</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">class</span> <span class="title class_">MapInfoAdmin</span>(admin.ModelAdmin):</span><br><span class="line">    <span class="comment"># 设置 列表页 显示字段名字的</span></span><br><span class="line">    list_display = [<span class="string">&#x27;id&#x27;</span>, <span class="string">&#x27;label&#x27;</span>, <span class="string">&#x27;name&#x27;</span>, <span class="string">&#x27;cdi&#x27;</span>, <span class="string">&#x27;hlrr&#x27;</span>, <span class="string">&#x27;mrcr&#x27;</span>, <span class="string">&#x27;speed&#x27;</span>, <span class="string">&#x27;rrsdr&#x27;</span>]</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment"># 注册模型类 到 admin站点里面去</span></span><br><span class="line">admin.site.register(MapInfo, MapInfoAdmin)</span><br><span class="line"></span><br><span class="line">admin.site.site_header = <span class="string">&#x27;全国主要城市交通数据&#x27;</span></span><br><span class="line">admin.site.site_title = <span class="string">&#x27;全国主要城市交通排行榜&#x27;</span></span><br><span class="line">admin.site.index_title = <span class="string">&#x27;欢迎查看交通数据&#x27;</span></span><br></pre></td></tr></table></figure>

<p>再刷新<code>127.0.0.1:8000/admin</code>发现有变化</p>
<h1 id="2-高德地图-中国主要城市交通健康榜-爬虫项目"><a href="#2-高德地图-中国主要城市交通健康榜-爬虫项目" class="headerlink" title="2.高德地图-中国主要城市交通健康榜-爬虫项目"></a>2.高德地图-中国主要城市交通健康榜-爬虫项目</h1><h2 id="2-1-数据总入口"><a href="#2-1-数据总入口" class="headerlink" title="2.1.数据总入口"></a>2.1.数据总入口</h2><p>找到爬取的根目录<code>https://report.amap.com/ajax/getCityRank.do</code></p>
<h2 id="2-2-数据分类"><a href="#2-2-数据分类" class="headerlink" title="2.2.数据分类"></a>2.2.数据分类</h2><p>城市编号label、城市名字name、拥堵延时指数cdi、高延时运行时间占比hlrr、拥堵路段里程比mrcr、平均车速speed、道路运行速度偏差率rrsdr</p>
<h2 id="2-3-scrapy基本使用"><a href="#2-3-scrapy基本使用" class="headerlink" title="2.3.scrapy基本使用"></a>2.3.scrapy基本使用</h2><h3 id="2-3-1-scrapy项目过程"><a href="#2-3-1-scrapy项目过程" class="headerlink" title="2.3.1.scrapy项目过程"></a>2.3.1.scrapy项目过程</h3><ul>
<li>创建项目： <code>scrapy startproject CityTraffic</code></li>
<li>进入项目： <code>cd CityTraffic</code></li>
<li>创建爬虫： <code>scrapy genspider city report.amap.com</code></li>
<li>运行爬虫： <code>scrapy crawl city</code></li>
</ul>
<h3 id="2-3-2-文件说明"><a href="#2-3-2-文件说明" class="headerlink" title="2.3.2.文件说明"></a>2.3.2.文件说明</h3><p>book.py 放置爬虫代码<br>items.py 编写字段key<br>middlewares.py 中间件，暂时用不到<br>pipelines.py 管道，存放数据库存储<br>settings.py 配置文件  </p>
<h3 id="2-3-3-完善项目"><a href="#2-3-3-完善项目" class="headerlink" title="2.3.3.完善项目"></a>2.3.3.完善项目</h3><ul>
<li><p>修改settings.py的君子协议，否则抓取不到重要数据<br><code>ROBOTSTXT_OBEY = False</code></p>
</li>
<li><p>修改settings.py的浏览器配置为自己网页的请求头：<br><code>USER_AGENT = &#39;Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36&#39;</code></p>
</li>
<li><p>修改city.py文件,修改的url为爬取网页的地址：</p>
<figure class="highlight ini"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">allowed_domains</span> = [<span class="string">&#x27;report.amap.com&#x27;</span>]</span><br><span class="line"><span class="attr">start_urls</span> = [<span class="string">&#x27;https://report.amap.com/ajax/getCityRank.do&#x27;</span>]</span><br></pre></td></tr></table></figure></li>
</ul>
<h3 id="2-3-4-数据入库"><a href="#2-3-4-数据入库" class="headerlink" title="2.3.4.数据入库"></a>2.3.4.数据入库</h3><ul>
<li><p>1.配置mysql<br>打开settings.py文件，新增代码：</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># mysql数据库配置</span></span><br><span class="line">MYSQL_HOST = <span class="string">&#x27;localhost&#x27;</span></span><br><span class="line">MYSQL_DB_NAME = <span class="string">&#x27;book&#x27;</span></span><br><span class="line">MYSQL_PORT = <span class="number">3306</span></span><br><span class="line">MYSQL_USER = <span class="string">&#x27;root&#x27;</span></span><br><span class="line">MYSQL_PASSWORD = <span class="string">&#x27;root&#x27;</span></span><br></pre></td></tr></table></figure>
</li>
<li><p>2.配置piplines.py文件</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># -*- coding: utf-8 -*-</span></span><br><span class="line"><span class="keyword">from</span> CityTraffic <span class="keyword">import</span> settings</span><br><span class="line"><span class="keyword">import</span> pymysql</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">class</span> <span class="title class_">CitytrafficPipeline</span>(<span class="title class_ inherited__">object</span>):</span><br><span class="line">    <span class="comment"># 1.python和mysql服务建立关系</span></span><br><span class="line">    <span class="keyword">def</span> <span class="title function_">open_spider</span>(<span class="params">self, spider</span>):</span><br><span class="line">        <span class="comment"># 链接成功</span></span><br><span class="line">        self.connc = pymysql.Connect(</span><br><span class="line">            host=settings.MYSQL_HOST,</span><br><span class="line">            database=settings.MYSQL_DB_NAME,</span><br><span class="line">            port=settings.MYSQL_PORT,</span><br><span class="line">            user=settings.MYSQL_USER,</span><br><span class="line">            password=settings.MYSQL_PASSWORD,</span><br><span class="line">            charset=<span class="string">&#x27;utf8&#x27;</span>,</span><br><span class="line">        )</span><br><span class="line">    </span><br><span class="line">        <span class="comment"># 创建游标对象</span></span><br><span class="line">        self.cur = self.connc.cursor()</span><br><span class="line"></span><br><span class="line">    <span class="comment"># 2.插入数据</span></span><br><span class="line">    <span class="keyword">def</span> <span class="title function_">process_item</span>(<span class="params">self, item, spider</span>):</span><br><span class="line"></span><br><span class="line">        <span class="keyword">try</span>:</span><br><span class="line">            <span class="comment"># 1.写 sql语句</span></span><br><span class="line">            sql = <span class="string">&#x27;insert into map_mapinfo values(%s,%s,%s,%s,%s,%s,%s,%s);&#x27;</span></span><br><span class="line"></span><br><span class="line">            <span class="comment"># 格式处理，把字典类型变为列表</span></span><br><span class="line">            items_list = <span class="built_in">list</span>(item.values())</span><br><span class="line">            <span class="comment"># 主键占位</span></span><br><span class="line">            items_list.insert(<span class="number">0</span>, <span class="number">0</span>)</span><br><span class="line">            <span class="built_in">print</span>(items_list)</span><br><span class="line"></span><br><span class="line">            <span class="comment"># 2.让游标执行 Sql</span></span><br><span class="line">            self.cur.execute(sql, items_list)</span><br><span class="line">            <span class="comment"># 事务提交</span></span><br><span class="line">            self.connc.commit()</span><br><span class="line">        <span class="keyword">except</span> Exception <span class="keyword">as</span> e:</span><br><span class="line">            <span class="built_in">print</span>(e)</span><br><span class="line">            <span class="comment"># 事务回滚</span></span><br><span class="line">            self.connc.rollback()</span><br><span class="line"></span><br><span class="line">        <span class="keyword">return</span> item</span><br><span class="line"></span><br><span class="line">    <span class="comment"># 3.爬虫结束--断开和mysql的连接</span></span><br><span class="line">    <span class="keyword">def</span> <span class="title function_">close_spider</span>(<span class="params">self, spider</span>):</span><br><span class="line">        self.cur.close()</span><br><span class="line">        self.connc.close()</span><br></pre></td></tr></table></figure></li>
</ul>
<h3 id="2-3-5-爬取数据"><a href="#2-3-5-爬取数据" class="headerlink" title="2.3.5.爬取数据"></a>2.3.5.爬取数据</h3><p>city.py的代码如下：</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># -*- coding: utf-8 -*-</span></span><br><span class="line"><span class="keyword">import</span> scrapy</span><br><span class="line"><span class="keyword">import</span> requests</span><br><span class="line"><span class="keyword">import</span> re</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">class</span> <span class="title class_">CitySpider</span>(scrapy.Spider):</span><br><span class="line">    name = <span class="string">&#x27;city&#x27;</span></span><br><span class="line">    allowed_domains = [<span class="string">&#x27;report.amap.com&#x27;</span>]</span><br><span class="line">    start_urls = [<span class="string">&#x27;https://report.amap.com/ajax/getCityRank.do&#x27;</span>]</span><br><span class="line"></span><br><span class="line">    <span class="keyword">def</span> <span class="title function_">parse</span>(<span class="params">self, response</span>):</span><br><span class="line">        content = response.text</span><br><span class="line">        pattern = <span class="string">r&#x27;[&#123;|&#125;]&#x27;</span></span><br><span class="line">        <span class="comment"># 以&#123;&#125;为分隔符将文本切割为列表</span></span><br><span class="line">        cont = re.split(pattern, content)</span><br><span class="line">        <span class="keyword">for</span> r <span class="keyword">in</span> cont:</span><br><span class="line">            city_name = re.sub(<span class="string">r&#x27;[A-Z|a-z|0-9|\[|\]|\,|\&#x27;|\&quot;|\:|\.|\-]&#x27;</span>, <span class="string">&#x27;&#x27;</span>, r)  <span class="comment"># 获得城市名字</span></span><br><span class="line">            <span class="comment"># re.sub(&#x27;[\u4e00-\u9fa5]&#x27;)正则表达式提取中文</span></span><br><span class="line">            city = &#123;&#125;</span><br><span class="line">            number = re.sub(<span class="string">&#x27;[\u4e00-\u9fa5:a-zA-Z\&quot;-]&#x27;</span>, <span class="string">&#x27;&#x27;</span>, r)</span><br><span class="line">            <span class="comment"># print(number)</span></span><br><span class="line">            label = re.search(<span class="string">&#x27;\\d&#123;6&#125;&#x27;</span>, number)  <span class="comment"># 搜索城市编号</span></span><br><span class="line">            <span class="keyword">if</span> label:</span><br><span class="line">                city[<span class="string">&#x27;label&#x27;</span>] = label.group()</span><br><span class="line">                city[<span class="string">&#x27;name&#x27;</span>] = city_name</span><br><span class="line">                dym = <span class="string">&#x27;https://report.amap.com/ajax/cityDaily.do?cityCode=&#x27;</span></span><br><span class="line">                index = <span class="string">&#x27;&amp;dataType=&#x27;</span></span><br><span class="line">                url = dym + city[<span class="string">&#x27;label&#x27;</span>] + index</span><br><span class="line">                <span class="comment"># 发送请求获取这个城市的详细信息,调用parse_map()函数</span></span><br><span class="line">                <span class="keyword">yield</span> scrapy.Request(url, callback=self.parse_map, meta=&#123;<span class="string">&#x27;map&#x27;</span>: city&#125;)</span><br><span class="line">            <span class="keyword">else</span>:</span><br><span class="line">                <span class="keyword">pass</span></span><br><span class="line"></span><br><span class="line">    <span class="keyword">def</span> <span class="title function_">parse_map</span>(<span class="params">self, response</span>):</span><br><span class="line">        <span class="comment"># 取字典</span></span><br><span class="line">        city = response.meta.get(<span class="string">&#x27;map&#x27;</span>)</span><br><span class="line">        <span class="comment"># 从reponse提取要解析的地址</span></span><br><span class="line">        <span class="comment"># url1 = response.url + &#x27;1&#x27;</span></span><br><span class="line">        url2 = response.url + <span class="string">&#x27;2&#x27;</span></span><br><span class="line">        url3 = response.url + <span class="string">&#x27;3&#x27;</span></span><br><span class="line">        url4 = response.url + <span class="string">&#x27;4&#x27;</span></span><br><span class="line">        url5 = response.url + <span class="string">&#x27;5&#x27;</span></span><br><span class="line">        <span class="comment"># 将解析网页的内容格式化</span></span><br><span class="line">        <span class="comment"># arr_cdi = requests.get(url1).json()</span></span><br><span class="line">        arr_hlrr = requests.get(url2).json()</span><br><span class="line">        arr_mrcr = requests.get(url3).json()</span><br><span class="line">        arr_speed = requests.get(url4).json()</span><br><span class="line">        arr_rrsdr = requests.get(url5).json()</span><br><span class="line">        <span class="comment"># cdi = </span></span><br><span class="line">        hlrr = mrcr = speed = rrsdr = <span class="number">0</span></span><br><span class="line">        <span class="comment"># 不同城市的rrsdr的道路数量不同，用size记录</span></span><br><span class="line">        size = <span class="built_in">len</span>(arr_rrsdr)</span><br><span class="line">        <span class="keyword">for</span> i <span class="keyword">in</span> <span class="built_in">range</span>(<span class="number">7</span>):</span><br><span class="line">            <span class="comment"># cdi += arr_cdi[i][1]  # cdi有的城市有问题，为爬取所有城市的数据，暂时注释掉</span></span><br><span class="line">            hlrr += arr_hlrr[i][<span class="number">1</span>]</span><br><span class="line">            mrcr += arr_mrcr[i][<span class="number">1</span>]</span><br><span class="line">            speed += arr_speed[i][<span class="number">1</span>]      </span><br><span class="line">        <span class="keyword">for</span> i <span class="keyword">in</span> <span class="built_in">range</span>(size):</span><br><span class="line">            rrsdr += arr_rrsdr[i][<span class="number">1</span>]</span><br><span class="line">        <span class="comment"># 将double转为str，否则无法存入数据库</span></span><br><span class="line">        <span class="comment"># city[&#x27;cdi&#x27;] = str(cdi/7)</span></span><br><span class="line">        city[<span class="string">&#x27;hlrr&#x27;</span>] = <span class="built_in">str</span>(hlrr/<span class="number">7</span>)</span><br><span class="line">        city[<span class="string">&#x27;mrcr&#x27;</span>] = <span class="built_in">str</span>(mrcr/<span class="number">7</span>)</span><br><span class="line">        city[<span class="string">&#x27;speed&#x27;</span>] = <span class="built_in">str</span>(speed/<span class="number">7</span>)</span><br><span class="line">        city[<span class="string">&#x27;rrsdr&#x27;</span>] = <span class="built_in">str</span>(rrsdr/size)</span><br><span class="line">        <span class="keyword">yield</span> city</span><br><span class="line"></span><br></pre></td></tr></table></figure>

<p>最后执行<code>scrapy crawl city</code>，爬取的数据入库  </p>
<figure class="highlight vim"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">cd</span> ..</span><br><span class="line"><span class="keyword">cd</span> GaodeMap</span><br><span class="line"><span class="keyword">python</span> manage.<span class="keyword">py</span> runserver</span><br></pre></td></tr></table></figure>
<p>启动django服务，打开网址<code>127.0.0.1:8000/admin</code>登陆后即可查看数据库内的内容</p>
<hr>

        
            <div class="donate-container">
    <div class="donate-button">
        <button id="donate-button">赞赏</button>
    </div>
    <div class="donate-img-container hide" id="donate-img-container">
        <img id="donate-img" src="" data-src="/myblog/img/donate.jpg">
        <p> 感谢鼓励 </p>
    </div>
</div>
        
        <br />
        <div id="comment-container">
        </div>
        <div id="disqus_thread"></div>
        <div id="lv-container"></div>
        <div class="giscus"></div>
    </div>
</div>

    </div>
</div>


<footer class="footer">
    <ul class="list-inline text-center">
        
        

        

        

        
        <li>
            <a target="_blank"  href="https://github.com/samuel-24276">
                            <span class="fa-stack fa-lg">
                                <i class="iconfont icon-github"></i>
                            </span>
            </a>
        </li>
        

        

    </ul>
    
    <p>
        <span>/</span>
        
        <span><a target="_blank" rel="noopener" href="https://github.com/samuel-24276">samuel-24276&#39;s Page</a></span>
        <span>/</span>
        
        <span><a target="_blank" rel="noopener" href="https://beian.miit.gov.cn/">北京-未备案</a></span>
        <span>/</span>
        
        <span><a href="#">返回顶部</a></span>
        <span>/</span>
        
    </p>
    
    <p>
        <span id="busuanzi_container_site_pv">
            <span id="busuanzi_value_site_pv"></span>PV
        </span>
        <span id="busuanzi_container_site_uv">
            <span id="busuanzi_value_site_uv"></span>UV
        </span>
        Created By <a target="_blank" rel="noopener" href="https://hexo.io/">Hexo</a>  Theme <a target="_blank" rel="noopener" href="https://github.com/aircloud/hexo-theme-aircloud">AirCloud</a></p>
</footer>




</body>

<script>
    // We expose some of the variables needed by the front end
    window.hexo_search_path = "search.json"
    window.hexo_root = "/myblog/"
    window.isPost = true
</script>
<script src="https://cdn.bootcss.com/jquery/3.3.1/jquery.min.js"></script>

<script src="/myblog/js/index.js"></script>

<script async src="//busuanzi.ibruce.info/busuanzi/2.3/busuanzi.pure.mini.js"></script>






    <script type="text/javascript">
       (function() {
           if (typeof LivereTower === 'function') { return; }

           var j, d = document.getElementById('lv-container');

           d.setAttribute('data-id','city');
           d.setAttribute('data-uid' , 'MTAyMC80OTA5OC8yNTU5Mw==');

           j = document.createElement('script');
           j.src = 'https://cdn-city.livere.com/js/embed.dist.js';
           j.async = true;

           d.appendChild(j);
       })();
    </script>
    <noscript>为正常使用来必力评论功能请激活JavaScript</noscript>
    </div>

</html>
